Many types of software have been identified in the field of software engineering, though no single taxonomy system is yet universally accepted. It is common to refer to types of software according to programming and architectural paradigms known in the art. In an article entitled “Language and Architecture Paradigms as Object Classes: A Unified Approach Towards Multiparadigm Programming” by Spinellis, et al., published in Programming Languages and System Architectures International Conference, pages 191–207, by Springer-Verlag in 1994, and which is incorporated herein by reference, the authors define a paradigm as “a category of entities sharing a common characteristic.” In the context of software, Spinellis, et al. note that “[c]omputer language paradigms offer linguistic abstractions . . . for expressing program implementations. Similarly, system architectures offer the hardware abstractions and quantitative theories applicable to the execution of compiled programs.” The authors go on to describe a number of properties which may be used to characterize different paradigms, e.g., the property of problem decomposition entity. The problem decomposition entity is the primary architectural entity in a paradigm, i.e., the principal unit used to characterize and implement an application. In the case of an object-oriented system, for example, objects comprise the primary architectural element, and are used to model and implement the software.
Object-oriented, procedural, logical, component-oriented, visual, agent-oriented, and concurrent software are only a few examples of architectural and programming paradigms known in the art. Moreover, because many of the paradigms are overlapping and independent, a software application may be described in terms of a plurality of paradigms. For example, a given application may be at once object-oriented, visual, concurrent, and component-based.
In an article titled “Toward an Execution Model for Component Software,” by Michael Franz, published in Special Issues in Object Oriented Programming: Workshop Reader of the 10th European Conference in Object-Oriented Programming ECOOP '96, pages 144–149, Linz, 1996, and which is incorporated herein by reference, the author states that “[w]hen discussing different programming paradigms, we usually base our taxonomy on the underlying design methodology, or, less subtly, on the programming language being used in a design. Strangely enough, this is no longer true when it comes to component-oriented programming.” Franz further suggests that the programming model, distribution model, and execution model are useful properties in the classification of types of software including component-oriented software.
Software designers typically select one or more programming and architectural paradigms in the design and implementation of an application. Many criteria exist in the art for selecting a paradigm, including compatibility with an existing system, cost of development and testing, cost of long-term maintenance, and similarity of the paradigm to the underlying application problem. As Spinellis, et al. observe “[i]t is widely accepted that each paradigm offers a different set of tradeoffs between efficiency, provability, elision, and implementation cost.”
A software paradigm identified in the art is message-oriented software. Message-oriented software treats messages as the primary architectural element, or the problem decomposition entity, according to Spinellis, et al. Applications are defined in terms of message flows, i.e., incoming and outgoing messages, and the processing performed on the messages. While the message-oriented paradigm is applicable to a wide range of business problems, the paradigm is especially compatible with a class of software applications known in the art as message-oriented middleware (MOM).
In an article titled “Message-Oriented Middleware” by Vondrak, published in the Carnegie-Mellon Software Technology Review which can be found at http://www.sei.cmu.edu/str/descriptions, and which is incorporated herein by reference, MOM is defined as “a client/server infrastructure that increases the interoperability, portability, and flexibility of an application by allowing the application to be distributed over multiple heterogeneous platforms. It reduces the complexity of developing applications that span multiple operating systems and network protocols by insulating the application developer from the details of the various operating system and net interfaces. Application Programming Interfaces (APIs) that extend across diverse platforms and networks are typically provided by the MOM.” In other words, MOM is software that focuses on the interface between applications in a distributed environment, possibly on a variety of platforms. MOM implements inter-application communication as a separate function, independent of the communicating applications. Thus, in a world of diverse systems, developed in different languages, using different paradigms, and running on different hardware, MOM facilitates interaction among the diverse systems. This capability is especially significant for organizations with a varied collection of legacy systems; further, it provides a way to integrate new systems, e.g., a system acquired as a result of a merger or acquisition, without substantial investment. A plurality of implementations of MOM exist in the art, for example, the MQSeries produced by IBM Corporation of Armonk, N.Y., and Tuxedo by BEA Systems, Inc. of San Jose, Calif.
FIG. 1 is a schematic block diagram depicting an architecture of a system 10 employing message-oriented middleware, as is known in the art. System 10 comprises client and/or server hardware platforms 12, 18, and 24, chosen from hardware platforms known in the art, and also designated respectively herein as client platform A, client platform B, and server platform C. Client platform A runs an application 14, also designated herein as client program A, and client platform B runs an application 20, designated herein as client program B. An application 26, also designated herein as server program C, runs on server platform C.
Client programs typically implement system functions visible to a user of the system, e.g., displaying results of a query or recording a new transaction. Server programs comprise programs providing services to client programs. Server programs supply a range of services, including access to a centralized resource such as a database, calculations, and communications services.
Client program A, client program B, and server program C are implemented using one or more of a plurality of programming and architectural paradigms known in the art. Client program A, client program B, and client program C may each utilize different architectural and programming paradigms. Typically, client program A, client program B, and server program C interact with each other by sending and receiving messages via a network 30. Client programs A, B, and C, and their respective platforms, are assumed to comprise different software paradigms and hardwares.
Because of the diversity of hardware platforms and software paradigms included in system 10, messages created by client program A must undergo transformations in order to render them understandable to network 30, and to client program B and server program C. Similar transformations are required for messages to and from client program B and server program C. A message-oriented middleware 16 provides the transformation service. MOM 16 comprises a platform interface which converts messages created by client program A from client platform A format into a protocol suitable for network 30, and transforms messages received by client program A from network 30 format into client platform A format. MOM 16 performs substantially similar transformations for client program B and server program C. Typically, the platform interface interprets message header and trailer data, removes or adds headers and trailers, and breaks up or reassembles messages.
In addition to the platform interface, MOM 16 comprises application-level logic for message processing. Thus, MOM 16 transforms a message sent by client program A, and produces a message that is valid in the context of communication via network 30; however, the message format and content may not match the expectations of one or more intended recipients of the message. Moreover, client program A is not necessarily aware of all the programs wishing to receive its message. For these cases, MOM 16 encapsulates the application logic required to re-format and route the message.
IBM's MQSeries Integrator comprises an example of an implementation of MOM 16. As described in Chapter 1 of MQSeries Integrator Introduction and Planning, published by IBM Corporation, 2000, and incorporated herein by reference, the services provided include the ability to “route a message to several destinations, using rules that act on the contents of one or more of the fields in the message or message header, transform a message, so that applications using different formats can exchange messages in their own formats . . . modify the contents of a message (for example, by adding data extracted from a database . . . )” Further, Chapter 1 cites the two principal benefits of MOM: message and data transformations are performed in a single place, thereby facilitating upgrades and modifications; and additional systems can be integrated quickly, by focusing on interface requirements.
Development of message-oriented software comprises defining message flows. Message flows describe the transformations, processing, and routing that are associated with a message. In an article titled “Coverage Analysis for Message Flows,” by Aizenbud-Reshef, to be presented at ISSRE 2001, the 12th International Symposium on Software Reliability Engineering in Hong Kong, Nov. 27–30, 2001, which is incorporated herein by reference, the author states that a message flow is a visual program describing the processing logic as a directed graph, which represents that processing performed on a message. Development environments for message-oriented software typically provide graphical tools for the definition and debugging of message flows, since visual programming is considered the most natural paradigm for message-oriented software. The Free On-Line Dictionary of Computing (FOLDOC), which can be found at http://foldoc.doc.ic.ac.uk/foldoc and which is incorporated herein by reference, defines a visual programming environment as “[s]oftware which allows the use of visual expressions (such as graphics, drawings, animation or icons) in the process of programming. These visual expressions may be used as graphical interfaces for textual programming languages. They may be used to form the syntax of new visual programming languages leading to new paradigms such as programming by demonstration or they may be used in graphical presentations of the behaviour or structure of a program.”
FIG. 2 is a schematic message flow directed graph illustrating a message flow 50, as is known in the art. Message flow 50 comprises messages, connections, and nodes. An incoming message 52 enters message flow 50 via an input node 56. An outgoing message 66 exits message flow 50 via an output node 60. Incoming message 52 passes through a number of nodes: input node 56, a processing node 58, and output node 60. Input node 56 and ouput node 60 handle the queuing functions required to receive or dispatch a message, and typically comprise implementations of message queues Processing node 58 applies logic and functions to the message. All nodes comprise zero or more terminals, through which messages enter and exit the node. For example, processing node 58 comprises four terminal: an input terminal 57, and three output terminals designated 59, 61, and 63.
Typically, a message flow developer works in a visual development environment and constructs message flows by selecting nodes from a store of pre-defined nodes, extending existing nodes to create new nodes, or creating completely new nodes. Message flow developers implement logic in nodes by writing code in a language such as Structured Query Language (SQL). A node may itself contain a message flow, termed a sub-flow, in which case the node is considered a compound node. In message flow 50, processing node 58 can identify a run-time exception, which directs the message flow to a special type of output terminal, a failure terminal 63 and an exception handler 62.
Message flows can range from the very simple, performing just one action on a message, to the complex, providing a number of actions on the message to transform its format and content. A message flow can process one message in several ways to deliver a number of output messages, perhaps with different format and content, to a number of target applications.
Developers test all types of software for the same reason: to provide an assurance that the software performs as specified, without defects. Testing, according to the Free On-Line Dictionary of Computing, is defined as “[t]he process of exercising a product to identify differences between expected and actual behaviour.” Since testing is costly and time-consuming, criteria are needed to assess the sufficiency of a set of tests. In an article titled “Test Adequacy Criteria for Component-Based Visual Software” by Keren and Shaham-Gafni, published in the Sixteenth International Conference on Testing Computer Software (TCS 99) in June, 1999, which is incorporated herein by reference, the authors note that “[t]he main technique for demonstrating that testing has been thorough is called test coverage analysis. Simply stated, the idea of test coverage analysis is to create, in some systematic fashion, a large and comprehensive list of tasks corresponding to the program under test. The tester's goal is to create a test suite that covers this list of tasks.”
FIG. 3 is a flowchart of elements and processes involved in a process 80 for testing software under test (SUT) 51, as is known in the art. Initially, in a determine coverage model step 82, a coverage model is chosen. A number of coverage models are known in the art, referring to different ways of assessing adequacy of testing. For example, statement coverage considers a percentage of program statements executed over a test suite, functional coverage involves measuring a percentage of specified functions that a test suite exercised, and path coverage concerns how many different control-flow paths the tests caused SUT 51 to execute. In an establish coverage goals step 84, goals are identified according to the coverage model chosen in step 82, the goals both directing the creation of a test suite and forming stopping criteria for overall testing process 80. Coverage goals comprise metrics referring to the extent to which SUT 51 is exercised, for example, a first metric has a goal of 95% statement coverage; a second metric has a goal of 100% functional coverage. In a define coverage tasks step 86, a set of coverage tasks is generated to meet the coverage goals identified in step 84. The coverage tasks comprise a translation of the coverage goals into practical terms relative to SUT 51. For example, a coverage goal of 95% statement coverage engenders a plurality of coverage tasks of the form “execute statement #n,” where n is a number comprising all statement numbers in SUT 51.
In a build test suite step 88, a test suite is generated, comprising a plurality of test conditions 90 and a way of evaluating expected results 92. Test conditions 90 comprise input values and conditions intended to exercise SUT 51 and perform one or more coverage tasks 86. Ideally, test conditions 90 should perform all coverage tasks generated in step 86, although in practice it may be difficult to identify a complete set of test conditions a priori. An oracle function typically evaluates expected results, either via manual inspection by a human expert or via automatic comparison to predetermined expected results 92. In an execution step 94, a test harness loads a test condition from test conditions 90 and executes SUT 51. Ideally, the test harness provides a way to execute SUT 51 while controlling inputs and monitoring outputs. During execution, a measure coverage step 96 runs, to assess the coverage achieved during the test. Execution of SUT 51 produces actual results, responsive to the test conditions loaded from test conditions 90. The oracle function performs a comparison step 98 between actual results of execution 94 and expected results 92, and a condition 100 determines the success or failure of the test. An outcome of failure generally indicates a defect in SUT 51, which requires developer attention in a debug step 102. A condition 104 checks whether sufficient testing of SUT 51 has completed, i.e., if the results of measuring coverage in step 96 accord with the coverage goals established in step 84. If the coverage goals have been accomplished, testing process 80 terminates.
If the coverage goals have not yet been achieved, testing process 80 continues in a condition 106 which checks if unexecuted tests remain in test suite 88. If tests remain, a next test condition is selected from test conditions 90, and execution step 94 again executes SUT 51 under the next test condition. If all tests in test suite 88 have executed without achieving the coverage goals of step 84, it is necessary to augment test suite 88 with additional tests in an add tests to test suite step 108, and continue in execution step 94.
Test adequacy criteria for different types of software have been proposed. For example, in Keren et el. propose coverage criteria for component-based visual software, based on utilization of a family of graph representations including the component containment graph, component call graph, component control flow graph, and framed graph. In an article titled “Testing Strategies for Form-Based Visual Programs” by Rothermel, et al., published in the Proceedings of the Eighth International Symposium on Software Reliability Engineering, pages 96–107, November 1997, and which is incorporated herein by reference, the authors suggest some test adequacy criteria for testing form-based programs, e.g., spreadsheet applications. The criteria make use of a cell relation graph, and include node and edge, cell dependence, and dataflow adequacy criteria.
Adequacy criteria are known in the art for testing object-oriented software. For example, in an article titled “Investigating the Applicability of Traditional Test Adequacy Criteria for Object-oriented Programs” by Kim et al., presented at FESMA-AEMES 2000: The European Software Measurement Conference, Madrid, October 2000, which is incorporated herein by reference, the authors explore the applicability of non-object-oriented adequacy criteria such as control-flow coverage and data flow-coverage to object-oriented software. Kim et al. conclude that “[t]he experimental result shows that the test criteria are able to deal with most of the OO [object-oriented] features, i.e., many of the defects related to the features could be revealed by the adequate test sets. However, they do not appear to be sufficient. There are a few OO-specific features that the investigated criteria are ineffective at handling, which indicates that the effectiveness of the criteria may be weakened in the presence of those OO features.” In sum, it is accepted in the art that test adequacy criteria, i.e., coverage criteria, must suit the software and architectural paradigms used in the application.